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The  dependencies  in  complex  software  systems  are  stretching  industry 
software  test  capabilities  such  that  schedules  and  budgets  are  constantly 
being  compromised  at  the  risk  of  producing  software  with  more  defects 
and  reliability  issues.  STAT  technologies  are  being  integrating  within  our 
production  code  to  facilitate  improved  testability  and  reliability.  Modeled 
from  techniques  utilized  in  hardware  systems  commonly  titled  Built-in  Test 
(BIT),  STAT  is  used  to  develop  applications  that  support  testability  without 
compromising  schedule  and  budget.  The  authors  have  found  that  STAT 
supports  the  development  and  deployment  of  robust  software  applications. 

•  Non-intrusive  test  techniques 

•  Improved  reliability 

•  Supports  test  automation 

•  Improves  integration  success 


The  purpose  of  this  briefing  is  to  provide  an  understanding  of 
what  a  Software  Test  Appliance  is  and  how  it  can  be  applied  to 
building  robust  and  reliable  Software  Applications. 

•  Background 

•  Example  Software  Systems 

•  Non-intrusive  test  techniques 

•  Improved  reliability 

•  Supports  test  automation 

•  Improves  integration  success 
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25+  years  in  Industry 
Emphasis  on  Software  Engineering 
Embedded  systems  for  Medical  applications 
Embedded  systems  for  Aircraft  avionics. 
Numerical  models  for  University  Research 
High  speed  measurement  tools 
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Mission  Planning  Software 

Flight  Performance  Planning  Models 

Embedded  Flight  Performance  Models  for 
Mission  Execution 
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Flight  Performance  Models  -  Digital  models  of  aircraft  performance 
flight  capabilities  and  limitations 

Developed  by  combining  flight-test  and  engineering  data  with 
standard  mathematical  models  (equations)  of  aircraft  performance 

Rotary  Aircraft  -  Performance  Planning  Cards  (PPC) 

Fixed  Wing  Aircraft  -  Take  Off  and  Landing  Data  (TOLD) 

-  Desktop  and  Onboard  Embedded  Applications 

-  Platforms 

•  Apache 

•  Blackhawk 

•  Chinook 

•  Kiowa  warrior 

•  JCA 

•  Sherpa 

•  Citation 


■9  TAKEOFF  AND  LANDING  DATA  CARD 
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Port+lblo  Flight 
Performance  System 


DERI  VIP 
Best  Range 
Best  Endurance 
Service  Ceifing 
Max  Airspeed 
Max  Gross  Weight 
etc. 


Performance 
Planning  Card 


On -Board  Performance/ 
Mission  Planning 


Real  Time  Map-  Owrlays 


Flight  Performance 
Model  integrated 
with  desktop 
applications 

•  Integrated 
Performance  and 
Aircraft  Configuration 
(IPAC) 

•  Portable  Flight 
Planning  Software 
(PFPS) 

•  Aviation/Joint  Mission 
Planning  System 
(AMPS/JMPS) 

Falconview 

Operators  Manual 
Charts 
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-  Embedded  Flight  Performance  Models  (EFPM)  are  being  inserted 
into  fixed  wing  and  rotary  wing  cockpits 

-  Enables  onboard  flight  performance  and  mission  planning 

-  Accurate  and  consistent  with  desktop  applications 

-  Very  fast  execution,  very  small  memory  requirements 

-  Efficient  numerical  methods  to  create  faster  EFPMs 
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•  Non-dimensional  data 

•  Higher-order  interpolation 

•  Pre-processing 

-  Currently  have  EFPM  onboard 

•  OH-58D 

•  UH-60M 

•  CH-47F 

•  MH-47G 

•  MH-60K/L/M 
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Today’s  systems  are  becoming  systems  of  Software  Systems 


Standardized  interfaces 
Abstracted  capabilities 
Functions 

-  Producer 

-  Consumer 

-  Mediators 
Coupling  Complexity 

-  Data 

-  Control 
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Data  coupling  -  The  dependence  of  a 
software  component  on  data  not  exclusively 
under  the  control  of  that  software 
component.  (DO-178B) 

-  Define  the  behavior  for  the  input  domain 

-  Bounds  check  the  component 

•  Outside  the  bounds 

•  At  the  bounds 

•  The  entire  domain 
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Control  coupling  -  The  manner  or  degree  by 
which  one  software  component  influences  the 
execution  of  another  software  component. 
(DO-178B) 

-  Requirements  must  fully  specify  switches 

-  Test  must  exercise  each  switch 

-  Reduce  control  coupling,  reduce  test  cases 
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•  Prevent  bugs  in  software  and  hardware 
before  deployment. 


•  Discover  symptoms  of  bugs  before  they 
affect  safety  or  functionality  of  systems. 

•  Provide  diagnostic  information  on  detected 


bugs. 
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Goals  of  testing  listed  above  p/as: 


Non-intrusive  verification  of  proper  functionality 
of  systems  for  operational  system  status. 

Non-intrusive  data  collection  to  support  the 
verification  of  required  operation. 


ion  of  changes  in  SW  configurations  and 
Jon  during  power  up  and  normal 
tions. 


Detec 
opera 
operal 

Provide  a  mechanism  to  detect  changes  in  data 

Provide  a  mechanism  to  support  developmental 
unit  test 


Provid 

valida 


e  a  mechanism  to  support  verification 
ion,  and  qualification  test 
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Techniques  designed  for  hardware 
systems  can  be  adapted  for  software 
systems 

•  Software  Built-in  Test  (SW-BIT) 

•  Interface  Logger  (SW-INF) 


Software  Engineering  Specialist 

»  www.crafidesigns.net 


•  Startup  BIT 


-  Evaluation  of  key  functions  and  capabilities  before 
transitioning  to  operational  system  status. 


•  Continuous  BIT 

-  Evaluation  of  selected  capabilities  during 
operational  system  status. 
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•  Initiated  BIT 

-  Detailed  BIT  used  to  provide  diagnostic  information 
while  temporarily  transitioned  to  non-operational 
system  status. 

•  Maintenance  BIT 

-  Exhaustive  BIT  designed  to  operate  with  a 
maintenance  interface  and  provide  “peek  and  poke” 
capabilities  into  system  during  both  operational  and 
non-operational  system  status. 


~4-.  n  i 
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Usually  performed  by  the  Boot  Loader  software. 

Evaluates  memory  locations  using  Destructive  Stuck- 
on-1/Stuck-on-0  tests  (memory  is  erased  before 
download  operations  start). 

Downloads  and  verifies  the  operational  code  using 
sequence  checking  and  check  summing  of  the 
operational  code. 

Activates  all  interfaces  and  verifies  that  they  are 
operational  by  receiving/sending  heartbeat 
messages. 

Activates  operational  code  and  verifies  when  it  is 
running. 


CMMI-ML3 


(  CDI  }  Software  Engineering  Specialist 

>  www.crafidesigns.net 


•  Foreground  Tests: 

-  Inputs: 

•  Checksum,  parity  checks,  time  tags,  sequence  numbers,  and 
heartbeat  checks  of  digital  and  discrete  inputs. 

•  Voltage,  current,  frequency  checks  of  analog  and  power  inputs. 

-  Processors 

-  Software 

•  Background  Tests: 

-  Inputs: 

•  Perform  loopback  tests  of  digital,  discrete,  and  analog  input. 

•  Non-destructive  Stuck-on-1/Stuck-on-0  tests  on  interface 
buffers. 

-  Memory: 

•  Non-destructive  Stuck-on-1/Stuck-on-0  tests  of  all  memory 


locations. 
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Detailed  evaluations  that  may  replace  Startup  BIT 
when  adequate  startup  testing  is  too  time- 
consuming. 

May  be  performed  by  operational  code,  however, 
IBIT  is  not  performed  during  normal  operation. 

Supports  maintenance  by: 

-  Identifying  where  problems  exist  as  well  as  problem  types. 

-  Providing  an  interface  for  maintenance  software  to  access 
memory  locations,  etc. 

-  Performing  download  evaluations. 
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Development  Platform: 


-  Provides  access  to  selected  memory  locations, 
by  setting  of  breakpoints,  etc.,  used  to  evaluate 
the  software  and/or  hardware. 


-  Sets  up  emulated/simulated  inputs  and  stimuli. 
Repair  Operations: 

-  Downloads  new  software  via  maintenance 
interfaces. 


-  Identifies  sources  of  problems  for  repair 
operations  on  LRU/SRUs. 

-  Evaluates  repair  status. 
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•  Operational  Evaluations  Only  -  Does  not  include  startup, 
development  and  V&V  evaluations/tests. 

•  Examples  of  Operational  Evaluations: 

-  Data  Analysis: 

•  Perform  sanity  checks  on  input  data. 

•  Prevent  run-time  errors  by  insuring  incorrect  and  out-of- 
bounds  data  are  not  used. 

-  Stack  Overflow  -  Provide  software  checks  to  insure 
against  and  report  conditions  where  stacks  overflow 
(especially  necessary  in  “C”,  C++,  and  other  languages). 

-  Exception  Handling  -  Provide  exception  handling 
capabilities  in  the  code  development. 
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-  Presence  of  data  sources 

-  Integrity  of  data  sources  (CRC) 


-  Use  of  correct  SW  computational  components  (compare 
computed  results  against  pre-computed  expected 
results) 

-  Expected  behaviors  from  selected  functional  SW 
components 

-  Correct  model  instantiation  (software  components, 
unique  parameters,  and  specific  data  sources) 
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•  Continuous  SW-BIT  checks: 

-  Integrity  of  data  sources  (CRC) 

-  Use  of  calculation  status  (NaN  &  status  flags) 

-  Memory  leaks 

-  Buffer  overruns 

-  Program  flow  by  choosing  test  cases  that  will 
to  maximize  code  coverage 


Initiated  SW-BIT  supports  regression 
testing: 


-  SW  tests  that  run  during  startup  and  under 
normal  operation  still  return  expected  results 


-  Test  case  stimulus  chosen  to  maximize  code 
coverage 

Maintenance  SW-BIT 


-Test  stubs 

-  Upgrade  verification/status 
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-  Computational  results 

-  Control  Flow 

-  Required  behaviors 

-  Hardware  and  software  system  configurations 

Flags  non-expected  results 
Supports  developmental  test 
Logs  test  case  stimulus  for  analysis 
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Include  in  operational  requirements 

Test  completely  to  avoid  false  failures 

Save  all  input  information 

Save  all  output  information 

Save  needed  state  information 

Strategically  capture  the  call  trace 

Develop  parsing  tools  to  support  analysis 

Identify  interfaces  where  data  can  easily 
be  gathered  without  intrusion. 
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•  Pick  a  simple  technique  that  will  not  be  accidently  enabled 


//  See  if  Test  Appliance  is  ready  to  be  enabled 

// 

//does  the  user  want  to  enable  logging? 

fopen_s (&logfile,  "c: \\{5432testapp-papalogl23}  \ 

\\eg45fhtymightbeagoodname . txt" ,  "r" ) ; 

if  (logfile  !=  NULL) 

{ 

logging  =  true; 
fclose (logfile) ; 


// 

//  Did  we  find  a  mode  to  calculate 

// 

if (i  <  pModel->cModes) 

{ 

if  (logging  ==  true  ) 

{ 

//  Write  State  info  and  input  info. 

//  Flush  the  file 
//  Close  the  File 

} 

ierr  =  pMyMode->prep  and  calc () ;  //  Calculate  the  mode 


if  (logging  ==  true) 

{ 

//  Write  the  output  info  and  state  info. 
//  flush  the  file 
//  close  the  file 
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+  [04/1 S/11] -16:20:56 
fi  [04/1 S/11]  -16:20:59 
E3  Inputs 

421  Anti-Ice:  0.000000 

1714  Blade  Erosion  Kit:  0.000000 

8269  Cabin  Doors:  0.000000 

8279  Cockpit  Doors:  0.000000 
1563  CONFIGURATION:  0.000000 

8280  Crew  Chief/Gunner  Windows:  0 

8267  ECS:  0.000000 

8270  ECS  State:  0.000000 

8263  EIBF:  0.000000 

8264  EIBF  Bypass  Doors:  0.000000 

1 772  Engine  #1  T orque  Factor:  1 .00L 

1 773  Engine  #2  T orque  Factor:  1 .00C 
8383  Engine  Type:  1.000000 

8273  Fixed/Additional  Download  Cha 
8756  Fixed/Additional  Drag:  0.00000( 

1 1 38  Free  Air  T emperature:  35.00000 
744  Heater:  0.000000 
1 295  IGE  Hover  Height:  1 0.000000 
8272  Internal/Additional  Load  Weight 

8265  IR  Suppression:  1.000000 

8274  Jettisonable  Stores  Download  C 
8023  Jettisonable  Stores  Drag:  0.0001 
736  Jettisonable  Stores  Weight:  0.00( 
8288  Max  Structural  Weight:  22000.0 

8268  OBOGS:  0.000000 

8271  OBOGS  State:  0.000000 

1 71 5  Operating  Limit  -  DE:  2.000000 
1560  Operating  Limit  -  SE:  3.000000 
1 955  0  perating  Weight:  1 2000. 0000C  i 

i 


r\. 

l  U  1  OU  1 

w  uv^rul 

ri-|-rn  n  _  l:  _ n  r-  _ 


n  rinrinn 


CORE::Min_Airspeed: 

IP:  1108 
+  Input  Array 
+  Accessories  Array 
+  Core  Values 
[±l  Core  Statuses 
EL  CORE::Convert_Airspeed: 

EL  CORE::Min_Airspeed: 

EL  CORE::Convert_Airspeed: 

3  Outputs 

730  Gross  Weight:  14412.000000 

855  MAX  ALLOWABLE  GWT  IGE:  22000.C 

856  MAX  ALLOWABLE  GWT  OGE:  20331. 
1718  MAX  HOVER  HEIGHT:  1000.000000 
1785  MAX  TORQUE  AVAILABLE  (Dual):  It 

1788  MAX  TORQUE  AVAILABLE  (Single  if 

1789  MAX  TORQUE  AVAILABLE  (Single  it: 

8307  MIN  SE  -  IAS  -W/O  STORES:  13.00( 

8308  MIN  SE  -  IAS  -W/STORES:  13.00001 
1324  PREDICTED  HOVER  TORQUE  (DE): 

1804  PREDICTED  HOVER  TORQUE  (SE# 

1805  PREDICTED  HOVER  TORQUE  (SE# 

1790  Total  Download  Change:  0.000000 
581  Total  Flat  Plate  Drag:  0.000000 
1180  TR  (Dual):  1.000000 

1791  TR  (Single  #1):  1.000000 

1792  TR  (Single  #2):  1.000000 
Status:  0 

3  Messages 

IPAC  Arrival 

.  T orque  Required  (1 1 1.0%)  exceeds  T orqui 
JDSLIST  1804 


J 


Log  File: 


jnjxj 


i.txt 


l 


Delete  Log 

i 

Parse  File 

Enable 

Logging 
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Program- 
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Cyclomatic  Graph 
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12 
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File: 

#  |  GO  | 

R 

1  CO 

1*  1  T  | 

T 

1  2 

;r*i  r  i 

F 

1  i 

CO  :  j<3 

Dec!  si  nn 

F3 

of  3:  K23I 

File: 

»  1  CO  1 

1  R 

1  CO 

1"  1  T  | 

1  T 

1  2 

2*1  K  | 

1  F 

1  1 

|C0  :  i<256 

Provides  call  trace 


Parser  can  flags  non-expected  results 
Supports  developmental  test 
Logs  test  case  stimulus  for  analysis 

Captured  data  can  be  fed  back  into 
application  as  a  stimulus  or  regression 


Provides  tangible  test  results 
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